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ABSTRACT 

The  recently  proposed  instantaneous  statistical  dimen¬ 
sion  is  compared  to  new  conditional  Renyi  entropies. 
The  motivation  for  introducing  these  time- varying  com¬ 
plexity  measures  is  the  analysis  of  electroencephalo¬ 
grams  for  which  nonstationarity  is  an  inherent  prop¬ 
erty.  Experimental  data  from  babies  are  analyzed  us¬ 
ing  the  proposed  complexity  measures.  The  instan¬ 
taneous  statistical  dimension  computation  is  based  on 
an  adaptive  autocorrelation  eigenspectrum  computa¬ 
tion  known  as  APEX  together  with  a  model  selection 
rule.  The  conditional  Renyi  entropies  are  based  on 
time-frequency  representation  of  the  signal.  It  is  shown 
that:  1)  the  three  time- varying  complexity  measures 
account  for  a  component  counting  property,  2)  the  in¬ 
stantaneous  statistical  dimension  is  the  most  robust  to 
Gaussian  white  noise. 

1.  INTRODUCTION 

An  electroencephalographic  signal  (EEG)  is  composed 
of  several  types  of  subsignals,  each  referring  to  a  par¬ 
ticular  state  of  the  brain.  The  complexity  of  EEG  sig¬ 
nals  originates  from  the  huge  number  of  degrees  of  free¬ 
dom  of  the  central  nervous  system.  Neurons  and  neu¬ 
ronal  networks  composing  the  central  nervous  system 
can  behave  in  an  asynchronous  or  synchronous  manner. 
Asynchronous  activity  leads  to  a  more  or  less  contin¬ 
uous  background  activity,  while  synchronous  activity 
leads  to  rhythmical  patterns  [1],  We  know  that  bio¬ 
logical  neurons  can  be  modeled  by  a  set  of  nonlinear 
differential  equations.  The  coupling  of  these  neurons 
results  in  a  complex  neuronal  network  that  can  ex¬ 
hibit  different  behaviour  like  spatio-temporal  patterns 
or  travelling  waves.  In  both  cases,  the  signal  recorded 
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at  a  particular  location  often  display  a  nonstationary 
behaviour  [1], 

Recently,  it  has  been  proposed  to  analyse  nonsta¬ 
tionary  time  series  with  an  adaptive  principal  compo¬ 
nent  neural  network  [2] .  A  linear  neural  network  is  used 
to  learn  and  track  the  eigenvalues  A (k)  —  {Af(&)|,  for 
i  —  1  7...7ns  and  associated  eigenvectors  e$(fe)  of  a 
time- varying  covariance  matrix  of  a  time  series  x(k). 
A  time-varying  model  selection  procedure  is  then  used 
to  select  the  most  relevant  eigenvalues  and  define  the 
instantaneous  statistical  dimension  (ISD)  m{k7T)  (T 
is  a  time-averaging  horizon).  It  is  shown  in  this  paper 
that  m(k7  T)  actually  count  the  number  of  signal  com¬ 
ponents  through  time.  An  other  time- varying  compo¬ 
nent  counting  methods  is  introduced  here.  It  is  based 
on  Renyi  conditional  entropies  computed  from  time- 
frequency  signal  representation.  Such  entropy  based 
time-frequency  information  measures  have  been  pro¬ 
posed  by  Williams  et  al  [3]  and  further  studied  by 
Flandrin  et  al  [4],  We  propose  in  this  paper  to  use 
conditional  entropies  to  compute  an  instantaneous  en¬ 
tropy  of  the  signal.  We  use  these  measures  to  analyse 
EEG  signals.  It  is  hoped  that  these  complexity  mea¬ 
sures  will  be  usefull  for  feature  extraction  in  EEG  sig¬ 
nal  processing  and  classification.  We  finally  show  that 
m{k7T)  can  be  used  to  segment  the  EEG  signal  and 
detect  rhytmical  activities.  The  recursive  nature  of  the 
algorithm  that  compute  m(k7  T)  makes  it  suitable  for 
on-line  segmentation. 

The  paper  is  organized  as  follows.  Section  2  de¬ 
scribes  the  experimental  set-up  used  for  recording  the 
electroencephalogram  (EEG).  Section  3  describes  the 
adaptive  method  to  estimate  the  eigenvalues  A (k)  and 
defines  the  ISD.  Section  4  presents  the  conditional  Renyi 
entropies.  Section  5  shows  some  numerical  examples 
and  Section  6  experiment  these  new  time- varying  com¬ 
plexity  measures  on  EEG  signals.  Section  7  concludes 
this  paper. 
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2.  EXPERIMENTAL  SETUP 


Five  to  twenty  EEG  signals  were  measured  on  four  ba¬ 
bies  using  electrodes  (Ag-AgCl  electrodes  flushed  with 
conductive  gel  and  adhered  by  tape)  attached  to  the 
skin  of  the  newborn.  The  electrode  placement  agrees 
with  the  American  EEG  Society  standards.  Slow  base¬ 
line  fluctuations  due  to  baby  movement  have  been  re¬ 
moved  by  using  a  2nd-order  high-pass  Butterworth  fil¬ 
ter  with  a  cut-off  frequency  of  0.1  Hz.  These  signals 
were  amplified  and  digitized  using  either  the  Amlab0 
or  Medelec0  (Oxford  Instruments)  Software/Hardware 
environment.  The  sampling  frequency  was  set  to  Fs  — 
256  Hz.  EEG  signals  were  then  sub-sampled  at  40  Hz. 

The  data  acquisitions  were  performed  at  the  Royal 
Women  Hospital  and  Royal  Children  Hospital,  Bris¬ 
bane.  Babies  were  showing  electrical  seizure  activity 
as  labelled  by  a  neurologist  from  the  Neurosciences  De¬ 
partment  at  the  Royal  Children’s  Hospital. 

3.  INSTANTANEOUS  STATISTICAL 
DIMENSION 

In  the  case  of  nonstationary  signals  such  as  the  EEG, 
an  adaptive  scheme  has  been  proposed  to  recursively 
compute  the  eigenvalue  spectrum  A*,  for  i  —  1, . . . ,  nS7 
of  the  autocovariance  matrix  Rxx(k7  r)  —  E[x(k)x(k  + 
r)\  of  a  digital  signal  x{k)  [5].  It  uses  linear  neural 
networks  with  Hebbian  learning  rule  to  estimate  the 
eigenvectors  and  eigenvalues.  The  output  yi(k)  of  the 
neural  network  at  the  node  i  at  time  k  is,  with  input 
vectors  x(&)  =  x/~  =  [x(i)x(i  +  1) . . .  x{i  +  ns  —  1]T 

yi-i(fe)  =  [p{(k)...pJ_1(k)]Tx(k) 

Di(k)  -  pf(fc)  x(fc)  +  Wf  yi-i(fc)  (1) 

the  MA  weight  vector  is  pf (k)  =  \pn  . .  -PmJ?  the  AR 
weight  vector  is  Wf (k)  =  [wn  . . .  wa- 1],  and  the  out¬ 
put  is  y f  (k)  —  [yi  . .  .yi].  From  equation  (1)  we  can 
see  that  pij  is  the  weight  connecting  input  x{k  +  i)  to 
the  output  yj ,  and  that  Wij  is  the  internal  weight  that 
connect  the  output  yj  to  the  output  yi.  Note  also  that 
the  network  architecture  is  nonsymmetrical  at  the  out¬ 
put  layer  W.  The  update  rule  for  the  weights  pi(k) 
and  W i(k)  is  the  following 

Apf(fc)  =  P  {(yi{k)  x{k)-yf{k)  p*(fe))}  (2) 

AW i{k)  =  -p  (yi{k)  yi-xi^+yiik)  W<(fe)) 

The  learning  rate  is  controlled  by  /?  which  can  be  es¬ 
timated  off-line  by  /?  =  min^l/Af},  or  on-line  by  the 
following  recursion  formula 


p(k)=p(k-l)/(1  +  p(k-l)y2(k))  (3) 

Using  equation  (3)  allows  us  to  have  an  adaptive  step 
size  for  each  node  i  in  the  network.  In  a  stationary 
environment,  the  output  yfik)  is  a  stochastic  estimate 
of  the  eigenvalue  A*.  It  is  expected  that  for  a  nonsta¬ 
tionary  signal  x{k)j  the  output  yj(k)  will  follow  the 
time- variation  of  the  ith.  eigenvalue  of  Rxx  (fe,  r)  with 
some  fluctuations  around  the  true  eigenvalues  due  to 
the  stochastic  nature  of  the  input  signal  x{k). 

Definition:  The  instantaneous  statistical  dimen¬ 
sion  m(k7T)  of  the  time  series  x{k)  at  time  k  for  a 
window  length  T  is  given  by 

m(k,T)  —  arg  min  MDL(y2  (k),i)  (4) 

iE{l,...,n8} 

where  yf(fe)  =  (1/T)  O')  and  ft.  (k)  = 

\y\(k)7 . . .  7y2n  ( k)]T  .  The  minimum  description  length 
[2]  MDL(A(k)7  i)  depends  on  the  values  y 2ns{k)  at  the 
time  instant  k.  The  MDL  function  also  depends  on  the 
number  of  samples  Nt  —  max {1/1  —  7 ,T}  taken  into 
account  in  the  adaptive  algorithm. 

Note  that  we  average  the  outpout  power  y\ (k)  over 
a  symmetric  time  interval  about  k.  The  time  averaged 
instantaneous  embedding  dimension  m(k7  T)  depends 
on  the  time  location  k  and  also  on  the  window  size  T. 

While  it  has  been  shown  that  there  is  no  simple 
relation  between  m{k7T)  and  the  dimension  of  the  un¬ 
derlying  dynamical  system,  m{k7T)  provides  us  with 
the  following  informations:  1)  if  m{k7T)  >  nW7  the 
signal  contains  a  nonstochastic  activity  of  high  dimen¬ 
sion.  This  could  reflects  asynchronous  behaviour  of  the 
brain  neurons.  If  1  <  m{k7T)  <  nW7  the  signal  con¬ 
tains  a  nonstochastic  activity  of  low  dimension.  This 
is  typical  from  synchronous  activity  of  the  neurons. 

4.  CONDITIONAL  RENYI  ENTROPIES 

An  interesting  attempt  to  measure  the  information  con¬ 
tent  in  time- varying  signal  x{t)  has  been  made  in  [3,4]. 
They  introduced  the  Renyi  entropies  H^(C(t7  /))  of 
time-frequency  distributions  (TFD)  of  Cohen’s  class 
C(t7f)  assuming  a  probability  density  function  (pdf) 
interpretation  of  C (t7  f) .  They  show  that  H ^  (C  ( t7f )) 
actually  count  the  number  of  component  in  a  signal. 
Unfortunately,  we  loose  the  time  location  aspect  when 
we  compute  the  scalar  value  H^\c(t7f)).  We  thus 
define  the  conditional  Renyi  entropies  of  a  normalized 
TFD  which  allows  to  introduce  instantaneous  Renyi 
entropies  and  thus  a  time-varying  component  count¬ 
ing  property.  Let  us  first  define  the  conditional  pdf 


C(f\t)  -  C(t,f)/C(t )  where  C(t )  =  fC(t,f)  df.  The 
conditional  Renyi  entropies  for  a  >  0  is 

H{n\C{f\t))  =  ^  log  2f  Ca(f\t )  df  (5) 

Note  that  (f\t))  — >  Hs(C(f\t))  which  is,  by 

definition,  the  conditional  Shannon  entropy.  We  can 
recover  the  entropy  H^\c(ty  /))  by 

Hr\c (t, f))  =  ^  log 2f 2 (i-)Ka)c(mca(t)  df 

(6) 

The  Renyi  entropies  H ^  ( C(t ,  /))  and  H ^  ( C(f\t )) 
have  units  of  bits.  The  Shannon  entropy  can  be  com¬ 
puted  Hs(C(ty  /))  =  /)).  The  entropies 

Hft\c(f\t))  depends  on  the  time  which  makes  them 
suitable  for  defining  time- varying  complexity  measures. 
Suitable  values  for  alpha  are  small  odd  integer  [4]  such 
as  a  —  3. 

5.  NUMERICAL  EXAMPLES 

In  order  to  validate  our  two  time-varying  complexity 
measure  and  assess  their  time  varying  component  count¬ 
ing  properties,  we  construct  the  following  signal  x(t)  — 
xi(t)+X2(t)+n(t)  where  07(f)  —  sin{ui{t)t)e~^t~ti^^2l‘iJ 
Ui(t)  —  2i +  otit)y  and  n(t)  is  a  Gaussian  White 
noise.  In  the  following  we  have  t\  —  10s,  £2  =  12s,  71  = 
6s,  72  =  3s,  f[o)  =  2 Hz,  /2(o)  =  3 Hz,  oi  =  0.15s“2, 
a2  —  0.2s-2.  The  signal  contains  1  component  up  to 
about  10s  and  then  one  more  component  is  added.  The 
TFD  used  in  these  examples  is  the  spectrogram,  and 
Figure  1  shows  the  spectrogram  of  the  signal  x(t)  to¬ 
gether  with  the  two  components. 
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Figure  1:  Spectrogram  of  x(t)  for  SNR  —  100 dB.  Time 
signal  and  power  spectral  density  functions  are  displayed  on 
left  and  bottom  of  the  figure.  Sampling  frequency,  window 
type  and  window  length  are  aslo  displayed  on  the  figure. 

Three  examples  are  displayed  in  Figure  2,  using 
SNR  —  100,20, 10 dB.  We  observe  that,  for  SNR  — 


100 dB,  Hft\c(f\t))  increases  by  1  bit  as  the  signal 
move  from  1  to  2  components.  We  also  observe  the  bor¬ 
der  effects  that  are  due  to  numerical  implementation  of 
the  spectrogram.  The  slow  transition  around  10s  is  due 
to  the  time-spreading  effect  in  using  the  spectrogram. 


Figure  2:  Conditional  Renyi  entropies  and  ISD  for  SNR  — 
100 dB  (solid  lines),  SNR  —  20 dB  (dotted- dashed  lines), 
and  SNR  —  10 dB  (dotted  lines) 

Figure  2  shows  the  great  robustness  against  noise 
(at  least  Gaussian  white  noise)  for  the  ISD.  Moreover, 
the  transition  region  around  the  10s  is  very  short.  The 
algorithm  show  very  fast  convergence  for  all  SNR.  The 
component  counting  property  is  illustrated  in  these  ex¬ 
amples. 

One  crutial  distinction  between  entropy-  and  ISD- 
based  complexity  measures  is  the  influence  of  noise.  As 
Hft\c(f\t))  is  computed  from  a  TFD  of  the  noisy  sig¬ 
nal,  this  quantity  will  be  highly  affected  by  the  later 
because  the  noise  spread  its  power  in  frequency  by 
essence.  On  the  other  hand,  the  ISD  measure  tend  to 
dissociate  the  deterministic  from  the  stochastic  compo¬ 
nents  and  provide  a  complexity  measure  of,  essentially, 
the  derterministic  part.  This  will  be  further  illustrated 
in  the  next  section. 

6.  EXPERIMENTAL  RESULTS 

The  two  major  biosignal  processing  applications  are 
detection  (or  segmentation)  and  feature  extraction  for 
classification  and  diagnosis.  The  goal  of  designing  these 
time- varying  complexity  measures  follows  exactely  these 
two  directions  when  dealing  with  nonstationary  signals. 
First  the  set  of  values  of  the  entropies  and  the  ISD  can 
be  used  in  their  own  as  features  of  the  EEG  signal.  Sec¬ 
ond,  using  thresholding  methods  we  can  use  them  to 
segment  the  EEG .  All  the  figures  3  to  5  shows  the  ISD 
(dotted-dashed  lines)  and  Hj^~3\c(f\t))  (solid  lines). 
Figure  3  shows  a  similar  decreasing  behavior  of  both 
complexity  measures. 

Figure  4  shows  an  almost  constant  conditional  en¬ 
tropy,  while  ISD  display  clearely  large  fluctuations  when 
EEG  rhytms  take  place.  Still,  carefull  observation  of 
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Figure  3:  ISD  and  H^=s) (C(f\t))  of  one  EEG  channel 

both  measures  reveal  some  positive  correlations.  In  this 
figure,  ISD  increased  during  the  main  ictal  segments. 
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Figure  4:  ISD  and  H^~3) of  one  EEG  channel 

Figure  5  displays  a  quite  constant  H^~3\C(f\t)), 
while  ISD  varies  considerably  when  EEG  rhytmical  ac¬ 
tivity  take  place.  In  this  figure,  the  ISD  tends  to  de¬ 
crease  during  the  ictal  segments  while  increases  in  the 
inter-ictal  ones.  This  does  not  contradict  the  two  pre¬ 
vious  results,  but  simply  emphasize  the  fact  that  ISD 
provide  a  complexity  measure  of  the  deterministic  part 
of  the  signals. 
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Figure  5:  ISD  and  H^~3) of  one  EEG  channel 

7.  CONCLUSIONS 

Physiological  measurement  such  as  EEG  display  some 
nonstationary  behaviour.  In  order  to  characterize  such 


nonstationary  mixture  of  deterministic  and  stochastic 
behaviour,  we  have  developed  two  time- varying  com¬ 
plexity  measures.  The  first  one  is  based  on  eigen¬ 
value  tracking  and  the  second  one  derived  from  the 
generalised  conditional  Renyi  entropies.  While  sharing 
the  same  component  counting  properties,  the  ISD  re¬ 
vealed  to  be  more  robust  in  noisy  measurements  such 
as  EEG  signals.  It  is  expected  that  these  new  concepts 
will  leads  to  new  and  interesting  tools  for  the  analy¬ 
sis  of  nonstationary  time  series.  Especially  in  the  case 
of  noisy  measurements,  MDL  selection  rule  has  been 
shown  to  perform  well  in  extracting  relevant  features 
in  the  signal.  Experimental  analysis  of  EEG  signals 
has  been  performed  and  the  time-varying  complexity 
analysis  reveals  some  interesting  insight  into  the  brain 
function. 
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